Modelling and managing reliability growth during the engineering design process

[This is a keynote speech presented at the 2nd International Conference on Design Engineering and Science, discussing modelling and managing reliability growth during the engineering process.] Reliability is vital for safe and efficient operation of systems. Decisions about the configuration and selection of parts within a system, and the development activities to prove the chosen design, will influence the inherent reliability. Modelling provides a mechanism for explicating the relationship between the engineering activities and the statistical measures of reliability so that useful estimates of reliability can be obtained. Reliability modelling should be aligned to support the decisions taken during design and development. We examine why and how a reliability growth model can be structured, the type of data required and available to populate them, the selection of relevant summary measures, the process for updating estimates and feeding back into design to support planning decisions. The modelling process described is informed by our theoretical background in management science and our practical experience of working with UK industry

Trading reliability targets within a supply chain using Shapley's value

The development of complex systems involves a multi-tier supply chain, with each organisation allocated a reliability target for their sub-system or component part apportioned from system requirements. Agreements about targets are made early in the system lifecycle when considerable uncertainty exists about the design detail and potential failure modes. Hence resources required to achieve reliability are unpredictable. Some types of contracts provide incentives for organisations to negotiate targets so that system reliability requirements are met, but at minimum cost to the supply chain. This paper proposes a mechanism for deriving a fair price for trading reliability targets between suppliers using information gained about potential failure modes through development and the costs of activities required to generate such information. The approach is based upon Shapley's value and is illustrated through examples for a particular reliability growth model, and associated empirical cost model, developed for problems motivated by the aerospace industry. The paper aims to demonstrate the feasibility of the method and discuss how it could be extended to other reliability allocation models

Expert Elicitation for Reliable System Design

This paper reviews the role of expert judgement to support reliability assessments within the systems engineering design process. Generic design processes are described to give the context and a discussion is given about the nature of the reliability assessments required in the different systems engineering phases. It is argued that, as far as meeting reliability requirements is concerned, the whole design process is more akin to a statistical control process than to a straightforward statistical problem of assessing an unknown distribution. This leads to features of the expert judgement problem in the design context which are substantially different from those seen, for example, in risk assessment. In particular, the role of experts in problem structuring and in developing failure mitigation options is much more prominent, and there is a need to take into account the reliability potential for future mitigation measures downstream in the system life cycle. An overview is given of the stakeholders typically involved in large scale systems engineering design projects, and this is used to argue the need for methods that expose potential judgemental biases in order to generate analyses that can be said to provide rational consensus about uncertainties. Finally, a number of key points are developed with the aim of moving toward a framework that provides a holistic method for tracking reliability assessment through the design process.Comment: This paper commented in: [arXiv:0708.0285], [arXiv:0708.0287], [arXiv:0708.0288]. Rejoinder in [arXiv:0708.0293]. Published at http://dx.doi.org/10.1214/088342306000000510 in the Statistical Science (http://www.imstat.org/sts/) by the Institute of Mathematical Statistics (http://www.imstat.org

Estimating rate of occurrence of rare events with empirical Bayes : a railway application

Classical approaches to estimating the rate of occurrence of events perform poorly when data are few. Maximum likelihood estimators result in overly optimistic point estimates of zero for situations where there have been no events. Alternative empirical-based approaches have been proposed based on median estimators or non-informative prior distributions. While these alternatives offer an improvement over point estimates of zero, they can be overly conservative. Empirical Bayes procedures offer an unbiased approach through pooling data across different hazards to support stronger statistical inference. This paper considers the application of Empirical Bayes to high consequence low-frequency events, where estimates are required for risk mitigation decision support such as as low as reasonably possible. A summary of empirical Bayes methods is given and the choices of estimation procedures to obtain interval estimates are discussed. The approaches illustrated within the case study are based on the estimation of the rate of occurrence of train derailments within the UK. The usefulness of empirical Bayes within this context is discusse

Historical Exploration - Learning Lessons from the Past to Inform the Future

This report examines a number of exploration campaigns that have taken place during the last 700 years, and considers them from a risk perspective. The explorations are those led by Christopher Columbus, Sir Walter Raleigh, John Franklin, Sir Ernest Shackleton, the Company of Scotland to Darien and the Apollo project undertaken by NASA. To provide a wider context for investigating the selected exploration campaigns, we seek ways of finding analogies at mission, programmatic and strategic levels and thereby to develop common themes. Ultimately, the purpose of the study is to understand how risk has shaped past explorations, in order to learn lessons for the future. From this, we begin to identify and develop tools for assessing strategic risk in future explorations. Figure 0.1 (see Page 6) summarizes the key inputs used to shape the study, the process and the results, and provides a graphical overview of the methodology used in the project. The first step was to identify the potential cases that could be assessed and to create criteria for selection. These criteria were collaboratively developed through discussion with a Business Historian. From this, six cases were identified as meeting our key criteria. Preliminary analysis of two of the cases allowed us to develop an evaluation framework that was used across all six cases to ensure consistency. This framework was revised and developed further as all six cases were analyzed. A narrative and summary statistics were created for each exploration case studied, in addition to a method for visualizing the important dimensions that capture major events. These Risk Experience Diagrams illustrate how the realizations of events, linked to different types of risks, have influenced the historical development of each exploration campaign. From these diagrams, we can begin to compare risks across each of the cases using a common framework. In addition, exploration risks were classified in terms of mission, program and strategic risks. From this, a Venn diagram and Belief Network were developed to identify how different exploration risks interacted. These diagrams allow us to quickly view the key risk drivers and their interactions in each of the historical cases. By looking at the context in which individual missions take place we have been able to observe the dynamics within an exploration campaign, and gain an understanding of how these interact with influences from stakeholders and competitors. A qualitative model has been created to capture how these factors interact, and are further challenged by unwanted events such as mission failures and competitor successes. This Dynamic Systemic Risk Model is generic and applies broadly to all the exploration ventures studied. This model is an amalgamation of a System Dynamics model, hence incorporating the natural feedback loops within each exploration mission, and a risk model, in order to ensure that the unforeseen events that may occur can be incorporated into the modeling. Finally, an overview is given of the motivational drivers and summaries are presented of the overall costs borne in each exploration venture. An important observation is that all the cases - with the exception of Apollo - were failures in terms of meeting their original objectives. However, despite this, several were strategic successes and indeed changed goals as needed in an entrepreneurial way. The Risk Experience Diagrams developed for each case were used to quantitatively assess which risks were realized most often during our case studies and to draw comparisons at mission, program and strategic levels. In addition, using the Risk Experience Diagrams and the narrative of each case, specific lessons for future exploration were identified. There are three key conclusions to this study: Analyses of historical cases have shown that there exists a set of generic risk classes. This set of risk classes cover mission, program and strategic levels, and includes all the risks encountered in the cases studied. At mission level these are Leadership Decisions, Internal Events and External Events; at program level these are Lack of Learning, Resourcing and Mission Failure; at Strategic Level they are Programmatic Failure, Stakeholder Perception and Goal Change. In addition there are two further risks that impact at all levels: Self-Interest of Actors, and False Model. There is no reason to believe that these risk classes will not be applicable to future exploration and colonization campaigns. We have deliberately selected a range of different exploration and colonization campaigns, taking place between the 15th Century and the 20th Century. The generic risk framework is able to describe the significant types of risk for these missions. Furthermore, many of these risks relate to how human beings interact and learn lessons to guide their future behavior. Although we are better schooled than our forebears and are technically further advanced, there is no reason to think we are fundamentally better at identifying, prioritizing and controlling these classes of risk. Modern risk modeling techniques are capable of addressing mission and program risk but are not as well suited to strategic risk. We have observed that strategic risks are prevalent throughout historic exploration and colonization campaigns. However, systematic approaches do not exist at the moment to analyze such risks. A risk-informed approach to understanding what happened in the past helps us guard against the danger of assuming that those events were inevitable, and highlights those chance events that produced the history that the world experienced. In turn, it allows us to learn more clearly from the past about the way our modern risk modeling techniques might help us to manage the future - and also bring to light those areas where they may not. This study has been retrospective. Based on this analysis, the potential for developing the work in a prospective way by applying the risk models to future campaigns is discussed. Follow on work from this study will focus on creating a portfolio of tools for assessing strategic and programmatic risk

To outsource or not to outsource!

In this article we will take a look at the phenomena of outsourcing as an overarching business concept that is, in short, about contracting of a specific bit of our business to a third part organisation. Consequently, outsourcing is a natural part of the make, share or buy continuum, as illustrated in Figure 1. We would, therefore, argue that outsourcing is not a new business phenomena as it has been commonly practiced since the early times of industrialisation, even though recently it has been enjoying renewed attention fuelled by the globalising forces

Applying Bayes linear methods to support reliability procurement decisions

Bayesian methods are common in reliability and risk assessment, however, such methods often demand a large amount of specification and can be computationally intensive. Because of this, many practitioners are unable to take advantage of many of the benefits found in a Bayesian-based approach. The Bayes linear methodology is similar in spirit to a Bayesian approach but offers an alternative method of making inferences. Bayes linear methods are based on the use of expected values rather than probabilities, and updating is carried out by linear adjustment rather than by Bayes Theorem. The foundations of the method are very strong, based as they are in work of De Finetti and developed further by Goldstein. A Bayes linear model requires less specification than a corresponding probability model and for a given amount of model building effort, one can model a more complex situation quicker. The Bayes linear methodology has the potential to allow us to build ''broad-brush' models that enable us, for example, to explore different test setups or analysis methods and assess the benefits that they can give. The output a Bayes linear model is viewed as an approximation to 'traditional' probabilistic models. The methodology has been applied to support reliability decision making within a current United Kingdom Ministry of Defence (MOD) procurement project. The reliability decision maker had to assess different contractor bids and assess the reliability merit of each bid. Currently the MOD assess reliability programmes subjectively using expert knowledge - for a number of reasons, a quantitative method of assessment in some projects is desirable. The Bayes linear methodology was used to support the decision maker in quantifying his assessment of the reliability of each contractor's bid and determining the effectiveness of each contractor's reliability programme. From this, the decision maker was able to communicate to the project leader and contractors, why a specific contractor was chosen. The methodology has been used in other MOD projects and is considered by those within the MOD as a useful tool to support decision making. The paper will contain the following. The paper will introduce the Bayes linear methodology and briefly discuss some of the philosophical implications of adopting a Bayes linear methodology within the context of a reliability programme analysis. The paper will briefly introduce the reliability domain and the reasons why it is believed that the Bayes linear methodology can offer support to decision makers. An in-depth analysis of the problem will then be given documenting the steps taken in the project and how future decision makers can apply the methodology. A brief summary will then be given as to possible future work for those interested in the Bayes linear methodology

Cost-benefit modelling for reliability growth

Decisions during the reliability growth development process of engineering equipment involve trade-offs between cost and risk. However slight, there exists a chance an item of equipment will not function as planned during its specified life. Consequently the producer can incur a financial penalty. To date, reliability growth research has focussed on the development of models to estimate the rate of failure from test data. Such models are used to support decisions about the effectiveness of options to improve reliability. The extension of reliability growth models to incorporate financial costs associated with 'unreliability' is much neglected. In this paper, we extend a Bayesian reliability growth model to include cost analysis. The rationale of the stochastic process underpinning the growth model and the cost structures are described. The ways in which this model can be used to support cost-benefit analysis during product development are discussed and illustrated through a simple case

A load sharing system reliability model with managed component degradation

Motivated by an industrial problem affecting a water utility, we develop a model for a load sharing system where an operator dispatches work load to components in a manner that manages their degradation. We assume degradation is the dominant failure type, and that the system will not be subject to sudden failure due to a shock. By deriving the time to degradation failure of the system, estimates of system probability of failure are generated, and optimal designs can be obtained to minimize the long run average cost of a future system. The model can be used to support asset maintenance and design decisions. Our model is developed under a common set of core assumptions. That is, the operator allocates work to balance the level of the degradation condition of all components to achieve system performance. A system is assumed to be replaced when the cumulative work load reaches some random threshold. We adopt cumulative work load as the measure of total usage because it represents the primary cause of component degradation. We model the cumulative work load of the system as a monotone increasing and stationary stochastic process. The cumulative work load to degradation failure of a component is assumed to be inverse Gaussian distributed. An example, informed by an industry problem, is presented to illustrate the application of the model under different operating scenarios

A model for availability growth with application to new generation offshore wind farms

A model for availability growth is developed to capture the effect of systemic risk prior to construction of a complex system. The model has been motivated by new generation offshore wind farms where investment decisions need to be taken before test and operational data are available. We develop a generic model to capture the systemic risks arising from innovation in evolutionary system designs. By modelling the impact of major and minor interventions to mitigate weaknesses and to improve the failure and restoration processes of subassemblies, we are able to measure the growth in availability performance of the system. We describe the choices made in modelling our particular industrial setting using an example for a typical UK Round III offshore wind farm. We obtain point estimates of the expected availability having populated the simulated model using appropriate judgemental and empirical data. We show the relative impact of modelling systemic risk on system availability performance in comparison with estimates obtained (Lesley Walls) from typical system availability modelling assumptions used in offshore wind applications. While modelling growth in availability is necessary for meaningful decision support in developing complex systems such as offshore wind farms, we also discuss the relative value of explicitly articulating epistemic uncertainties